Detect wedged vGPU VFs from the guest and report to the health store - #435

Summary

Top half of the wedged-VF work, stacked on #462 (the VF health store, placement exclusion, admission, and /resources fields). This PR adds the detection path that feeds that store.

• The guest agent watches /dev/kmsg for kernel-facility NVRM: ... RmInitAdapter failed! records, probes driver init at boot with nvidia-smi -L when the image has it, and exposes the current GPU initialization state over GetGPUInitStatus via vsock.
• A host sentinel controller polls vendor-VFIO instances with up to 64 concurrent checks and reports failures and successes into the VF health store from Quarantine unhealthy vGPU VFs via a persisted health store #462.
• Only instances with a live VMM (control socket present) are polled; a stopped or standby instance whose failed release left its claim in metadata is skipped, since QEMU vsock dials by guest CID alone and a stale CID could be reused by an unrelated instance.
• Reports are keyed on GPUClaimedAt from Quarantine unhealthy vGPU VFs via a persisted health store #462, the identity a claim receives when it is persisted.

Rebuilt on the claim-first allocator

This branch was rebuilt from the new #462 head after #321 merged as the claim-first rewrite. The guest agent, guest RPC, sentinel controller, wiring, metrics, and docs are the previously reviewed code, squashed into one commit. What changed in the port:

• GPUAssignedAt is gone from main; the sentinel keys reports on GPUClaimedAt, which Quarantine unhealthy vGPU VFs via a persisted health store #462 sets in the same metadata save as the claim.
• The GPURetainedForCleanup skip is gone with the retention-record design. The existing control-socket check already excludes instances without a VMM.
• The "allocate the vGPU immediately before metadata persistence" change to create.go is dropped. Claim-first already persists the claim before the VF is touched.
• The reconcile liveness change is re-hooked into main's two hypervisorMayBeAlive call sites; reconcile and the create/start cleanup guard share one wrapper that logs the resolution error and increments the liveness counter when it fails closed.
• The retention-record tests that the old branch adjusted no longer exist and were dropped with them.

Safety and failure handling

• GPU initialization success is terminal in the guest agent, so a delayed failure watcher cannot replace a confirmed success.
• The host trusts only the guest-agent RPC; workload output on the shared serial console cannot influence the tally.
• A slow nvidia-smi attempt is killed after 30 seconds. The guest agent waits for that process to be reaped before retrying, so an attempt stuck in uninterruptible I/O cannot accumulate concurrent probes; the independent kmsg watcher still reports the underlying init failure.
• Unreachable guests do not serialize a host-wide scan; checks run with a fixed concurrency limit.
• Repeated polls and controller restarts do not double-count an assignment. The sentinel revalidates the VF assignment before reporting, and the health store deduplicates reports by assignment.
• Reconciliation preserves claims when hypervisor liveness is uncertain and records that condition in logs and metrics.

Observability

• hypeman_instances_vgpu_sentinel_init_failures_total
• hypeman_instances_vgpu_sentinel_quarantines_total
• hypeman_instances_vgpu_sentinel_checks_total by result (ok, failed, unknown, rpc_error, unsupported_agent, or list_error)
• hypeman_instances_vgpu_quarantined_vfs
• hypeman_instances_vgpu_vf_health_store_unavailable
• hypeman_instances_vgpu_liveness_uncertain_total

lib/devices/GPU.md gains the detection and sentinel documentation.

Out of scope

An operator force-cycle endpoint. Recovery remains the documented manual DCGM quiesce, SR-IOV cycle, state edit, restart, and verification flow.

Testing

Passed locally:

go vet ./lib/instances ./lib/devices ./lib/guest ./lib/system/guest_agent ./lib/providers ./cmd/api
go test -race ./lib/devices ./lib/guest ./lib/system/guest_agent ./lib/resources ./lib/providers
go test -race ./lib/instances -run 'VGPU|Sentinel|Reconcile|Metric|HypervisorMayBeAlive|ResolveLiveHypervisor|Select|Quarantin|StoredVGPU|Cleanup|ReleaseStored|Snapshot.*GPU|Fork.*GPU'

cmd/api/wire_gen.go was regenerated with wire and matches. lib/guest/guest.pb.go and guest_grpc.pb.go are carried over from the previous head; guest.proto has not changed on main since, and the header records the same protoc and protoc-gen-go versions as the committed files.

The full lib/instances suite was not run here; tests that boot real VM images need a host this environment does not provide.

The underlying wedge signal and manual recovery sequence were previously validated on L40S hardware. A live end-to-end run of the guest watcher and host controller is still required before merge, including two failed assignments and the success path.